In this Jenkins tutorial, I explained the detailed steps to set up Jenkins master and scale Jenkins build agents on Kubernetes pods using Jenkins kubernetes plugin
In another post, I have explained how to setup Docker based Jenkins agents.
If you have a Kubernetes cluster in your environment, running Jenkins agents on the Kubernetes pods will give you good build isolation for different applications versions.
Also, an ephemeral Kubernetes pod based Jenkins agent is a great way of reducing the cost of the CI environment as Jenkins agents get spun up only if there is a build request.
How Does Jenkins Kubernetes Pod Agents Work?
Before getting into the implementation, let’s understand how this setup works.
- Whenever you trigger a Jenkins job, the Jenkins Kubernetes plugin will make an API call to create a Kubernetes agent pod.
- Then, the Jenkins agent pod gets deployed in the kubernetes with a few environment variables containing the Jenkins server details and secrets.
- When the agent pod comes up, it uses the details in its environment variables and talks back to Jenkins using the JNLP method. The following images show the environment variables of the agent pod.
All the build steps from the Jenkinsfile run on that pod. Once the build is complete, the pod will get terminated automatically. There are also options to retain the build pod.
The Jenkins Kubernetes plugin takes care of all the communication from Jenkins to the Kubernetes cluster.
Also, as long as your Kubernetes cluster scales, you can scale your Jenkins build agents without any issues.
Setting Up Jenkins Build Pods On Kubernetes
To work on this setup we need the following.
- A working Kubernetes cluster.
- Kubernetes admin user to create Kubernetes deployments and service accounts
- A running Jenkins master
Also, I am considering two scenarios here.
- Jenkins master running inside the Kubernetes cluster.
- Jenkins master running outside the Kubernetes cluster.
We will look at both scenarios and their configurations.
Overfall, here is what we are going to do.
- Create a namespace
devops-tools - Create a Kubernetes service account named
jenkins-adminwith permissions to manage pods indevops-toolsnamespace. This service account will be used by Jenkins to deploy the agent pods. (Both internal & external Jenkins) - Deploy Jenkins in
devops-toolsnamespace with thejenkins-adminservice account. (If you dont have an existing Jenkins) - Configure Kuberntes Jenkins Plugin for Jenkins to interact with Kubernetes cluster and deploy build agents.
Setting Up Kubernetes Namespace & Service Account
Let’s get started with the setup.
Step 1: Create a namespace called devops-tools
kubectl create namespace devops-tools
Step 2: Save the following manifest as service-account.yaml. It contains the role and role-binding for the service account with all the permission to manage pods in the devops-tools namespace.
apiVersion: v1
kind: ServiceAccount
metadata:
name: jenkins-admin
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: jenkins
namespace: default
labels:
"app.kubernetes.io/name": 'jenkins'
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: jenkins-role-binding
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: jenkins
subjects:
- kind: ServiceAccount
name: jenkins-admin
namespace: default
Create the service account.
kubectl apply -f service-account.yaml
Jenkins Master Setup in Kubernetes
In this setup, we will have both the Jenkins master and agents deploying in the same Kubernetes cluster.
We will set up the Jenkins master server on the Kubernetes cluster.
Note: If you have an existing setup, you can use that as well. Ensure it has a service account with permissions to deploy pods in the namespace where Jenkins is deployed.
Save the following manifest as deployment.yaml. This manifest contains persistent volume, deployment, and service definitions.
Note: Ensure that your Kubernetes cluster setup supports persistent volumes. If you deploy Jenkins without a persistent volume, you will lose the Jenkins data on every restart or pod deletion.
# Persistent Volume Claim
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-pv-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
# Deployment Config
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: jenkins-deployment
spec:
replicas: 1
selector:
matchLabels:
app: jenkins
template:
metadata:
labels:
app: jenkins
spec:
serviceAccountName: jenkins-admin
securityContext:
fsGroup: 1000
runAsUser: 1000
containers:
- name: jenkins
image: jenkins/jenkins:lts
resources:
limits:
memory: "2Gi"
cpu: "1000m"
requests:
memory: "500Mi"
cpu: "500m"
ports:
- name: httpport
containerPort: 8080
- name: jnlpport
containerPort: 50000
livenessProbe:
httpGet:
path: "/login"
port: 8080
initialDelaySeconds: 90
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 5
readinessProbe:
httpGet:
path: "/login"
port: 8080
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
volumeMounts:
- name: jenkins-data
mountPath: /var/jenkins_home
volumes:
- name: jenkins-data
persistentVolumeClaim:
claimName: jenkins-pv-claim
# Service Config
---
apiVersion: v1
kind: Service
metadata:
name: jenkins-service
annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: /
prometheus.io/port: '8080'
spec:
selector:
app: jenkins
type: NodePort
ports:
- name: httpport
port: 8080
targetPort: 8080
nodePort: 32000
- name: jnlpport
port: 50000
targetPort: 50000
Create the deployment.
kubectl apply -f deployment.yaml
After a couple of minutes, the Jenkins deployment will be up and you will be able to access over any Kubernetes node on port 32000
Step 4: Access the Jenkins dashboard over node port and unlock it using the password from the pod logs. Install the suggested plugins and create a Jenkins user.
Please follow the Jenkins on Kubernetes blog if you have any doubts.
Jenkins Kubernetes Plugin Configuration
Jenkins Kubernetes plugin is required to set up Kubernetes-based build agents. Let’s configure the plugin.
Step 1: Install Jenkins Kubernetes Plugin
Go to Manage Jenkins –> Manage Plugins, search for Kubernetes Plugin in the available tab and install it. The following Gif video shows the plugin installation process.